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Abstract 

We show that a large fraction of the data-structure lower bounds known today in fact follow by 
reduction from the communication complexity of lopsided (asymmetric) set disjointness. This includes 
lower bounds for: 

• high-dimensional problems, where the goal is to show large space lower bounds. 

• constant-dimensional geometric problems, where the goal is to bound the query time for space 
0(n ■ polylogn). 

• dynamic problems, where we are looking for a trade-off between query and update time. (In this 
case, our bounds are slightly weaker than the originals, losing a lglgn factor.) 

Our reductions also imply the following new results: 

• an fi(lgn/ lglgn) bound for 4-dimensional range reporting, given space 0(n ■ polylogn). This is 
quite timely, since a recent result [39] solved 3D reporting in 0(lg 2 lgn) time, raising the prospect 
that higher dimensions could also be easy. 

• a tight space lower bound for the partial match problem, for constant query time. 

• the first lower bound for reachability oracles. 

In the process, we prove optimal randomized lower bounds for lopsided set disjointness. 

1 Introduction 

The cell-probe model can be visualized as follows. The memory is organized into cells (words) of w bits each. 
A data structure occupies a space of S cells. The CPU receives queries and, for dynamic data structures, 
updates online. The CPU starts executing each individual operation with an empty internal state (no 
knowledge about the data structure), and can proceed by reading or writing memory cells. The running 
time is defined to be equal to the number of memory probes; any computation inside the CPU is free. 

The predictive power of the cell-probe model (stemming from its machine independence and information- 
theoretic flavor) have long established it as the de facto standard for data-structure lower bounds. The end 
of the 80s saw the publication of two landmark papers in the field: Ajtai's static lower bound for predecessor 
search pQ , and the dynamic lower bounds of Fredman and Saks [55] . In the 20 years that have passed, cell- 
probe complexity has developed into a mature research direction, with a substantial bibliography: we are 

aware of [D [H 131 GUI 133 m [3 123 133 El E3 E El ED [II [13 111 [13 133 ISI1 111 ill ED 111 1101113 11H]- 

The topics being studied cluster into three main categories: 



"The conference version of this paper appeared in FOCS'08 under the title (Data) Structures. 

tmipaalum.mit.edu. AT&T Labs. Parts of this work were done while the author was at MIT and IBM Almaden Research 
Center. 
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Figure 1: Dashed lines indicate reductions that were already known, while solid lines indicate novel reduc- 
tions. For problems in bold, we obtain stronger results than what was previously known. 



Dynamic problems. Here, the goal is to understand the trade-off between the query time t q , and the 
update time t u . The best known lower bound |41j implies that max{ i 9 ,t u } = f2(lgn). Most proofs employ a 
technique introduced by Fredman and Saks [26] , which divides the time line into "epochs" , and argues that 
a query needs to read a cell written in each epoch, lest it will miss an important update that happened then. 

High-dimensional static problems. These are "hard problems," exhibiting a sharp phase transition: 
either the space is linear and the query is very slow (e.g. linear search); or the space is very large (super- 
polynomial) and the query is essentially constant. 

Proofs employ a technique introduced by Miltersen [35j , which considers a communication game between 
a party holding the query, and a party holding the database. Simulating the CPU's cell probes, the querier 
can solve the problem by sending t q lg S bits. If we lower bound this communication by some a, we conclude 
that S > 2 n ( Q / t «). The bounds are interesting (often tight) for constant query time, but degrade quickly for 
higher t q . 

Low-dimensional static problems. These are problems for which we have polylogarithmic query bounds, 
with near linear space. The main research goal (within reach) has been to find the best query time for space 
S = 0(n ■ polylogn). The best known bound [13] implies that t q = fi(lgn/lglgn). The technique used in 
this regime, introduced by Patra§cu and Thorup |44j . is to consider a direct sum communication game, in 
which 0(n/polylogn) queriers want to communicate with the database simultaneously. 

The cross-over of techniques between the three categories has so far been minimal. At the same time, 
the diversity inside each category appears substantial, even to someone well- versed in the field. However, we 
will see that this appearance is deceiving. 

By a series of well-crafted reductions (Figure [1]), we are able to unify a large majority of the known 
results in each of the three categories. Since the problems mentioned in Figure [T] are rather well known, we 
do not describe them here. The reader unfamiliar with the field can consult Appendix El which introduces 
these problems and sketches some of the known reductions. 

All our results follow, by reductions, from a single lower bound on the communication complexity of 
lopsided (asymmetric) set disjointness. In this problem, Alice and Bob receive two sets S, respectively T, 
and they want to determine whether S D T = 0. The lopsided nature of the problem comes from the set 
sizes, \S\ <C |T|, and from the fact that Alice may communicate much less than Bob. 
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For several problems, our unified proof is in fact simpler than the original. This is certainly true for 2D 
range counting [40] , and arguably so for exact nearest neighbor [TT] and marked ancestor [5] (though for the 
latter, our bound is suboptimal by a lglgn factor). 

For 2D stabbing and 4D range reporting, we obtain the first nontrivial lower bounds, while for partial 
match, we improve the best previous bound [31] to an optimal one. 

It seems safe to say that the sweeping generality of our results come as a significant surprise (it has 
certainly been a major source of surprise for the author). A priori, it seems hard to imagine a formal 
connection between such lower bounds for very different problems, in very different settings. Much of the 
magic of our results lies in defining the right link between the problems: reachability queries in butterfly 
graphs. Once we decide to use this middle ground, it is not hard to give reductions to and from set 
disjointness, dynamic marked ancestor, and static 4-dimensional range reporting. Each of these reductions 
is natural, but the combination is no less surprising. 

1.1 New Results 

Partial match. Remember that in the partial match problem, we have a data base of n strings in {0, l} d , 
and a query string from the alphabet {0, 1, The goal is to determine whether any string in the database 
matches this pattern, where * can match anything. In §4j we show that: 

Theorem 1. Let Alice hold a string in {0,1, -k} d , and Bob hold n points in {0,l} d . In any bounded-error 
protocol answering the partial match problem, either Alice sends fl(d) bits or Bob sends f2(n 1_e ) bits, for 
any constant s > 0. 

By the standard relation between asymmetric communication complexity and cell-probe data struc- 
tures [36] and decision trees [7], this bounds implies that: 

• a data structure for the partial match problem with cell-probe complexity t must use space 2^ d / t \ 
assuming the word size is 0(n 1_£ /i). 

• a decision tree for the partial match problem must have size 2™ d > , assuming the depth is 0(n 1_2e /(i) 
and the predicate size is 0(n e ). 

As usual with such bounds, the cell-probe result is optimal for constant query time, but degrades quickly 
with t. Note that in the decision tree model, we have a sharp transition between depth and size: when the 
depth is O(n), linear size can be achieved (search the entire database). 

The partial match problem is well investigated [35] Q31 (S3 S3] ■ The best previous bound [21] for Alice's 
communication was Q(d/\gn) bits, instead of our optimal tt(d). 

Our reduction is a simple exercise, and it seems surprising that the connection was not established before. 
For instance, Barkol and Rabani [TT] gave a difficult lower bound for exact near neighbor in the Hamming 
cube, though it was well known that partial match reduces to exact near neighbor. This suggests that partial 
match was viewed as a "nasty" problem. 

By the reduction of [30] . lower bounds for partial match also carry over to near neighbor in i^, with 
approximation < 3. See [7] for the case of higher approximation. 

Reachability oracles. The following problem appears very hard: preprocess a sparse directed graph in 
less than n 2 space, such that reachability queries (can u be reached from u?) are answered efficiently. The 
problem seems to belong to folklore, and we are not aware of any nontrivial positive results. By contrast, 
for undirected graphs, many oracles are known. 

In Sj5] we show the first lower bound supporting the apparent difficulty of the problem: 

Theorem 2. A reachability oracle using space S in the cell probe model with w-bit cells, requires query time 
fi(lgn/lg^). 
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The bound holds even if the graph is a subgraph of a butterfly graph, and in fact it is tight for this 
special case. If constant time is desired, our bounds shows that the space needs to be n 1 " 1 ^ 1 -*. This stands 
in contrast to undirected graphs, for which connectivity oracles are easy to implement with 0{n) space and 
O(l) query time. Note however, that our lower bound is still very far from the conjectured hardness of the 
problem. 

Range reporting in 4D. Range reporting in 2D can be solved in O(lglgn) time and almost linear 
space [4]; see [45] for a lower bound on the query time. 

Known techniques based on range trees can raise [4] a <i-dimensional solution to a solution in d + 1 
dimensions, paying a factor 0(\gn/ lglgn) in time and space. It is generally believed that this cost for each 
additional the dimension is optimal. Unfortunately, we cannot prove optimal lower bounds for large d, since 
current lower bound techniques cannot show bounds exceeding i7(lg n/ lg lg n). Then, it remains to ask about 
optimal bounds for small dimension. 

Until recently, it seemed safe to conjecture that 3D range reporting would require r2(lgn/ lglgn) query 
time for space 0(n ■ polylogn). Indeed, a common way to design a d-dimensions static data structure is to 
perform a plane sweep on one coordinate, and maintain a dynamic data structure for d — 1 dimensions. The 
data structure is then made persistent, transforming update time into space. But it was known, via the 
marked ancestor problem [5], that dynamic 2D range reporting requires f2(lgn/ lglgn) query time. Thus, 
static 3D reporting was expected to require a similar query time. 

However, this conjecture was refuted by a recent result of Nckrich [39 from SoCG'07. It was shown that 
3D range reporting can be done in doubly-logarithmic query time, specifically t q = 0(lg 2 lgn). Without 
threatening the belief that ultimately the bounds should grow by 8(lgn/lglgn) per dimension, this positive 
result raised the intriguing question whether further dimensions might also collapse to nearly constant time 
before this exponential growth begins. 

Why would 4 dimensions be hard, if 3 dimensions turned out to be easy? The question has a simple, 
but fascinating answer: butterfly graphs. By reduction from reachability on butterfly graphs, we show in S}2] 
that the gap between 3 and 4 dimensions must be f2(lgn): 

Theorem 3. A data structure for range reporting in 4 dimensions using space S in the cell probe model 
with w-bit cells, requires query time f2(lgn/lg^). 

For the main case w = O(lgn) and S — n ■ polylogn, the query time must be 0(lg nj lg lg n). This is 
almost tight, since the result of Nekrich implies an upper bound of (3(lgn lglgn). 

Range stabbing in 2D. In fact, our reduction from reachability oracles to 4D range reporting goes 
through 2D range stabbing, for which we obtain the same bounds as in Theorem [3] There exists a sim- 
ple reduction from 2D stabbing to 2D range reporting, and thus, we recover our lower bounds for range 
reporting [40 , with a much simpler proof. 

1.2 Lower Bounds for Set Disjointness 

In the set disjointness problem, Alice and Bob receive sets S and T, and must determine whether SDT = 0. 
We parameterize lopsided set disjointness (LSD) by the size of Alice's set l^l = N, and B, the fraction 
between the universe and N. In other words, S,T C [N ■ B]. We do not impose an upper bound on the size 
of T, i.e. \T\ < N ■ B. 

Symmetric set disjointness is a central problem in communication complexity. While a deterministic 
lower bound is easy to prove, the optimal randomized lower bound was shown in the celebrated papers of 
Razborov [IS] and Kalyanasundaram and Schnitger [35], dating to 1992. Bar-Yossef et al. [TU] gave a more 
intuitive information-theoretic view of the technique behind these proofs. 

In their seminal paper on asymmetric communication complexity, Miltersen et al. [36j proved an (easy) 
deterministic lower bound for LSD, and left the randomized lower bound as an "interesting" open problem. 
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Figure 2: A butterfly with degree 2 and depth 4. 

In FOCS'06, we showed [5] a randomized LSD lower bound for the case when B > poly(iV). For such 
large universes, it suffices to consider an independent distribution for Alice's and Bob's inputs, simplifying 
the proof considerably. 

In this paper, we show how to extend the techniques for symmetric set disjointness to the asymmetric 
case, and obtain an optimal randomized bound in all cases: 

Theorem 4. Fix 5 > 0. In a bounded error protocol for LSD, either Alice sends at least SNlgB bits, or 
Bob sends at least NB 1 ^ ^ bits. 

The proof appears in $6] and is fairly technical. If one is only interested in deterministic lower bounds, 
the proof of Miltersen et al. [35] suffices; this proof is a one-paragraph counting argument. If one wants 
randomized lower bounds for partial match and near-neighbor problems, it suffices to use the simpler proof 
of [5], since those reductions work well with a large universe. Randomized lower bounds for reachability 
oracles and the entire left subtree of Figure Q] require small universes (B <§; N) , and thus need the entire 
generality of Theorem |4j 

Organization. The reader unfamiliar with our problems is first referred to Appendix |Al which defines all 
problems, and summarizes the known reductions (the dashed lines in Figured]). 

The remainder of this paper is organized as a bottom- up, level traversal of the tree in Figure [T] (We find 
that this ordering builds the most intuition for the results.) 

In S}2] we explain why butterfly graphs capture the structure hidden in many problems, and show reduc- 
tions to dynamic marked ancestor, and static 2D stabbing. 

In Sj3j we consider some special cases of the LSD problem, which are shown to be as hard as the general 
case, but are easier to use in reductions. Subsequently, 21 and SJS] reduce set disjointness to partial match, 
respectively reachability oracles. 

Finally, $6] gives the proof of our optimal LSD lower bound. 

2 The Butterfly Effect 

The butterfly is a well-known graph structure with high "shuffle abilities." The graph (Figure [2]) is specified 
by two parameters: the degree 6, and the depth d. The graph has d + 1 layers, each having b d vertices. 
The vertices on level are sources, while the ones on level d are sinks. Each vertex except the sinks has 
out-degree d, and each vertex except the sources has in-degree d. If we view vertices on each level as vectors 
in [b] d , the ed ges going out of a vertex on level i go to vectors that may differ only on the ith coordinate. 
This ensures that there is a unique path between any source and any sink: the path "morphs" the source 
vector into the sink node by changing one coordinate at each level. 

For convenience, we will slightly abuse terminology and talk about "reachability oracles for G," where 
G is a butterfly graph. This problem is defined as follows: preprocess a subgraph of G, to answer queries 
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of the form, "is sink v reachable from source u?" The query can be restated as, "is any edge on the unique 
source-sink path missing from the subgraph?" 

2.1 Reachability Oracles to Stabbing 

The reduction from reachability oracles to stabbing is very easy to explain formally, and we proceed to do 
that now. However, there is a deeper meaning to this reduction, which will be explored in §2.21 

Reduction 5. Let G be a butterfly with M edges. The reachability oracle problem on G reduces to 2- 
dimensional stabbing over M rectangles. 

Proof. If some edge of G does not appear in the subgraph, what source-sink paths does this cut off? Say the 
edge is on level i, and is between vertices (• • • , ft*, Wj+i, • • • ) and (• • • , v^, i>j+i, • • • ). The sources 
that can reach this edge are precisely (*,••• Vi, w^+i, ■ • • ), where * indicates an arbitrary value. The sinks 
that can be reached from the edge are (• • • , Uj-i, v'^-k, ■ ■ ■ ). The source-sink pairs that route through the 
missing edge are the Cartesian product of these two sets. 

This Cartesian product has precisely the format of a 2D rectangle. If we read a source vector (vi, . . . , Vd) 
as a number in base b with the most significant digit being Vd, the set of sources that can reach the edge is 
an interval of length 6 i_1 . Similarly, a sink is treated as a number with the most significant digit v\, giving 
an interval of length b d ~ % . 

For every missing edge, we define a rectangle with the source and sink pairs that route through it. Then, 
a sink is reachable from a source iff no rectangle is stabbed by the (sink, source) point. □ 

Observe that the rectangles we constructed overlap in complicated ways. This is in fact needed, because 
2-dimensional range stabbing with non-overlapping rectangles can be solved with query time 0(lg 2 lg«) [20] . 
As explained in Appendix [XI 2D range stabbing reduces to 2D range counting and 4D range reporting. 

2.2 The Structure of Dynamic Problems 

The more interesting reduction is to the marked ancestor problem. The goal is to convert a solution to the 
dynamic problem into a solution to some static problem for which we can prove a lower bound. 

A natural candidate would be to define the static problem to be the persistent version of the dynamic 
problem. Abstractly, this is defined as follows: 

input: an (offline) sequence of updates to a dynamic problem, denoted by tii, . . . , u m . 

query: a query q to dynamic problem and a time stamp t < m. The answer should be the answer to q if 
it were executed by the dynamic data structure after updates u\, . . . , u T . 

An algorithm result for making data structures persistent can be used to imply a lower bound for the 
dynamic problem, based on a lower bound for the static problem. The following is a standard persistence 
result: 

Lemma 6. // a dynamic problem can be solved with update time t u and query time t q , its (static) persistent 
version will have a solution with space 0(m ■ t u ) and query time 0(t q ■ lglg(m • t u )). 

Proof. We simulate the updates in order, and record their cell writes. Each cell in the simulation has a 
collection of values and timestamps (which indicate when the value was updated). For each cell, we build a 
van Emde Boas predecessor structure [50] over the time-stamps. The structures occupy 0(m ■ t u ) space in 
total, supporting queries in 0(lg \g{mt u )) time. To simulate the query, we run a predecessor query for every 
cell read, finding the last update that changed the cell before time r. □ 

Thus, if the static problem is hard, so is the dynamic problem (to within a doubly logarithmic factor). 
However, the reverse is not necessarily true, and the persistent version of marked ancestor turns out to be 
easy, at least for the incremental case. To see that, compute for each node the minimum time when it 
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becomes marked. Then, we can propagate down to every leaf the minimum time seen on the root-to-leaf 
path. To query the persistent version, it suffices to compare the time stamp with this value stored at the 
leaf. 

As it turns out, persistence is still the correct intuition for generating a hard static problem. However, 
we need the stronger notion of full persistence. In partial persistence, as seen above, the updates create a 
linear chain of versions (an update always affects the more recent version) . In full persistence, the updates 
create a tree of versions, since updates are allowed to modify any historic version. 

For an abstract dynamic problem, its fully-persistent version is defined as follows: 

input: a rooted tree (called the version tree) in which every node is labeled with a sequence of update 
operations. The total number of updates is m. 

query: a query q to the dynamic problem, and a node r of the version tree. The answer should be the 
answer to q if it were executed after the sequence of updates found on the path through the version 
tree from the root to r. 

Like the partially persistent problem, the fully persistent one can be solved by efficient simulation of the 
dynamic problem: 

Lemma 7. If a dynamic problem can be solved with update time t u and query time t q , the fully-persistent 
static problem has a solution with space 0(m ■ t u ) and query time 0(t q lglg(m • t u )). 

Proof. For each cell of the simulated machine, consider the various nodes of the version tree in which the 
cell is written. Given a "time stamp" (node) t, we must determine the most recent change that happened 
on the path from r to the root. This is the longest matching prefix problem, which is equivalent to static 
predecessor search. Thus, the simulation complexity is the same as in Lemma [5] □ 

We now have to prove a lower bound for the fully-persistent version of marked ancestor, which we 
accomplish by a reduction from reachability oracles in the butterfly: 

Reduction 8. Let G be a subgraph of a butterfly with M edges. The reachability oracle problem on G reduces 
to the fully-persistent version of the marked ancestor problem, with an input of O(M) offline updates. The 
tree in the marked ancestor problem has the same degree and depth as the butterfly. 

Proof. Our inputs to the fully-persistent problem have the pattern illustrated in Figure [3] At the root of 
the version tree, we have update operations for the leaves of the marked ancestor tree. If we desire a lower 
bound for the incremental marked ancestor problems, all nodes start unmarked, and we have an update for 
every leaf that needs to be marked. If we want a decremental lower bound, all nodes start marked, and all 
operations are unmark. 

The root has b subversions; in each subversion, the level above the leaves in the marked ancestor tree is 
updated. The construction continues similarly, branching our more versions at the rate at which level size 
decreases. Thus, on each level of the version tree we have b d updates, giving b d ■ d updates in total. 

With this construction of the updates, the structure of the fully persistent marked ancestor problem is 
isomorphic to a butterfly. Imagine what happens when we query a leaf v of the marked ancestor tree, at a 
leaf t of the version tree. We think of both v and t as vectors in [b] d , spelling out the root to leaf paths. The 
path from the root to v goes through every level of the version tree: 

• on the top level, there is a single version {t is irrelevant), in which v is updated. 

• on the next level, the subversion we descend to is decided by the first coordinate of t. In this subversion, 
w's parent is updated. Note that w's parent is determined by the first d — I coordinates of v. 

• on the next level, the relevant subversion is dictated by the first two coordinates of t. In this subversion, 
i>'s grandparent is updated, which depends on the first d— 2 coordinates of v. 

• etc. 
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Figure 3: (a) The marked ancestor problem, (b) An instance of fully-persistent marked ancestor. 



This is precisely the definition of a source-to-sink path in the butterfly graph, morphing the source into the 
sink one coordinate at a time. Each update will mark a node if the corresponding edge in the butterfly is 
missing in the subgraph. Thus, we encounter a marked ancestor iff some edge is missing. □ 

Let us see how Reduction [8] combines with Lemma [7] to give a lower bound for the dynamic marked 
ancestor problem. Given a butterfly graph with to edges, we generate at most to updates. From Lemma [3 
the space of the fully persistent structure is S = 0(m ■ t u ), and the query time 0{t q \g\g(mt q )), where 
t u and t q are the assumed running times for dynamic marked ancestor. If t u — polylogm, the space is 
S = O(mpolylogm). 

The lower bound for reachability oracles from Theorem [2] implies that for space O(mpolylogm), the 



query time must be f2( 



lg rn 



lg lg m ) 



But we have an upper bound of 0(t q lg lg(mt q )) for the query time, so 



) . This is weaker by a lg lg to factor compared to the original bound of [5] . 



3 Adding Structure to Set Disjointness 

Just as it is more convenient to work with Planar-NAE-3SAT that Circuit-SAT for showing NP-completeness, 
our reductions use two restricted versions of LSD: 

Blocked-LSD: The universe is interpreted as [N] x [B], and elements as pairs (u, v). It is guaranteed that 
(V)x € [N], S contains a single element of the form (x,*). 

2-Blocked-LSD: The universe is interpreted as [^] x [B] x [B\. It is guaranteed that for all x £ [^] and 
y€ [B], S contains a single element of the form (x,y,*) and a single element of the form (x,*,y). 
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It is possible to reanalyze the lower bound of and show directly that it applies to these restricted 
versions. However, in the spirit of the paper, we choose to design a reduction from general LSD to these 
special cases. 

Lemma 9. LSD reduces to Blocked-LSD by a deterministic protocol with communication complexity O(N). 

Proof. In the general LSD, Alice's set S might contain multiple elements in each block. Alice begins by 
communicating to Bob the vector (ci, . . . , cat), where c, denotes the number of elements in block i. The 
number of distinct possibilities for (ci, . . . ,cn) is P^ 1 ): so Alice needs to send O(N) bits (in a possibly 
non- uniform protocol). 

Now Bob constructs a set T' in which the i-th block of T is included Cj times; a block with a = is 
discarded. Alice considers a set S' in which block i gets expanded into Cj blocks, with one element from the 
original block appearing in each of the new blocks. We now have an instance of Blocked-LSD. □ 

Lemma 10. Blocked-LSD reduces to 2-Blocked-LSD by a deterministic protocol with communication com- 
plexity O(N). 

Proof. Consider B consecutive blocks of Blocked-LSD. Adjoining these blocks together, we can view the 
universe as a B x B matrix. The matrix has one entry in each column (one entry per block), but may have 
multiple entries per row. The protocol from above can be applied to create multiple copies of rows with 
more elements. After the protocol is employed, there is one element in each row and each column. Doing 
this for every group of B blocks, the total communication will be -g- ■ 0(B) = O(N). □ 

Since the lower bound for LSD says that Alice must communicate ui(N) bits, these reductions show that 
Blocked-LSD and 2-Blockcd-LSD have the same complexity. 

3.1 Reductions 

Before proceeding, we must clarify the notion of reduction from a communication problem to a data-structure 
problem. In such a reduction, Bob constructs a database based on his set T, and Alice constructs a set of k 
queries. It is then shown that LSD can be solved based on the answer to the k queries on Bob's database. 

When analyzing data structures of polynomial space or more, we will in fact use just one query (k = 1). 
If the data structure has size S and query time t, this reduction in fact gives a communication protocol for 
LSD, in which Alice communicates t lg S bits, and Bob communicates tw bits. This is done by simulating the 
query algorithm: for each cell probe, Alice sends the address, and Bob sends the content from his constructed 
database. At the end, the answer to LSD is determined from the answer of the query. 

If we are interested in lower bounds for space n 1+ °^ l \ note that an upper bound of \gS for Alice's 
communication no longer suffices, because S = 0(n 1+£ ) and S — 0(n) yield the same asymptotic bound. 
The work-around is to reduce to k parallel queries, for large k. In each cell probe, the queries want to read 
some k cells from the memory of size S. Then, Alice can send lg (, ) bits, and Bob can reply with k ■ w. 
Observe that lg ( fe ) <C fclg S, if k is large enough. 

4 Set Disjointness to Partial Match 



Reduction 11. Blocked-LSD reduces to one partial match query over n = N ■ B strings in dimension 
d = 0(N\gB). 

Proof. Consider a constant weight code <j> mapping the universe [B] to {0, If we use weight b/2, we have 
(b/2) = 2f2(b) codewords. Thus, we may set b = O(lgS). 

If S = {(1, si), . . . , (N, sn)}, Alice constructs the query string <f>(s\)4>(s2) • • ■ , i.e. the concatenation of 
the codewords of each Sj. We have dimension d — N ■ b — 0(N\gB). 



9 



For each point (x, y) e T, Bob places the string O^ 1 ' 6 4>{y) Qi N ~ x ) b in the database. Now, if (i, si) G T, 
the database contains a string with 4>{si) at position (i — 1)6, and the rest zeros. This string is dominated 
by the query, which also has 4>{si) at that position. On the other hand, if a query dominates some string in 
the database, then for some (i, s,-) € S and (i, j/) € T, (j>(si) dominates <fi(y). But this means Si = y because 
in a constant weight code, no codeword can dominate another. □ 

From the lower bound on Blocked-LSD, we know that in a communication protocol solving the problem, 
cither Alice sends Q(NlgB) bits, or Bob sends N ■ B 1 ^ 5 > n 1-5 bits. Rewriting this bound in terms of n 
and d, either Alice sends fl(d) bits, or Bob sends n 1_<5 bits, for constant S > 0. 

This implies that a data structure with query time t requires space 2 n ^ d / t \ as long as the word size is 
to < n 1 ^ 5 jt. It also implies that any decision tree of depth n 1 ' 8 needs to have size 2 n ( d /*\ 



5 Set Disjointness to Reachability Oracles 

Since we want a lower bound for near-linear space, we must reduce LSD to k parallel queries on the reacha- 
bility oracle. The entire action is in what value of k we can achieve. Note, for instance, that k = N is trivial, 
because Alice can pose a query for each item in her set. However, a reduction with k — N is also useless. 
Remember that the communication complexity of Alice is t ■ lg (^) > t lg ( N ^) ■ But LSD is trivially solvable 
with communication lg ( N jf) , since Alice can communicate her entire set. Thus, there is no contradiction 
with the lower bound. 

To get a lower bound on t, k must be made as small as possible compared to N. Intuitively, a source-sink 
path in a butterfly of depth d traverses d edges, so it should be possible to test d elements by a single query. 
To do that, the edges must assemble in contiguous source-sink paths, which turns out to be possible if we 
carefully match the structure of the butterfly and the 2-Blocked-LSD problem: 

Reduction 12. Let G be a degree-B butterfly graph with N non-sink vertices and N ■ B edges, and let d be 
its depth. 2-Blocked-LSD reduces to ^ parallel queries to a reachability oracle for a subgraph of G. 

Proof. Remember that in 2-Blocked-LSD, elements are triples (x,y,z) from the universe [^] x [B] x [B]. 
We define below a bijection between [^] x [B] and the non-sink vertices of G. Since (x, y) is mapped to a 
non-sink vertex, it is natural to associate (x, y, z) to an edge, specifically edge number z going out of vertex 
{x,y). 

Bob constructs a reachability oracle for the graph G excluding the edges in his set T. Then, Alice must 
find out whether any edge from her set S has been deleted. By mapping the universe [-jy] x [B] to the nodes 
carefully, we will ensure that Alice's edges on each level form a perfect matching. Then, her set of N edges 
form lj disjoint paths from sources to sinks. Using this property, Alice can just issue ^ queries for these 
paths. If any of the source-sink pairs is unreachable, some edge in S has been deleted. 

To ensure Alice's edges form perfect matchings at each level, we first decompose the non-sink vertices of 
G into ^ microsets of B elements each. Each microset is associated to some level i, and contains nodes of 
the form (• • • , Vi-i,*, fj+i, •) on level i. A value (x, y) is mapped to node number y in a microset identified 
by x (through some arbitrary bijection between [^] and microsets). 

Let (x, 1, z\), . . . , (x, B, zb) be the values in S that give edges going out of microset x. If the nodes 
of the microset are the vectors (• • • , i>j_i,*, i>i+i, ■), the nodes to which the edges of S go are the vectors 
(• • • ,Vi-i,Zj,Vi+i, •) on the next level, where j € [B\. Observe that edges from different microsets cannot 
go to the same vertex. Also, edges from the same microset go to distinct vertices by the 2-Blocked property: 
for any fixed x, the Zj's are distinct. Since all edges on a level point to distinct vertices, they form a perfect 
matching. □ 

Let us now compute the lower bounds implied by the reduction. We obtain a protocol for 2-Blockcd- 
LSD in which Alice communicates Hg(^) = 0(tklgj) = 0(N ■ |lg^) bits, and Bob communicates 
k ■ t ■ w = 0(N ■ j ■ w) bits. On the other hand, the lower bound for 2-Blocked-LSD says that Alice needs 
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to communicate Q(NlgB) bits, or Bob needs to communicate NB X ~ 5 , for any constant 5 > 0. It suffices to 
use, for instance, S = i. 

Comparing the lower bounds with the reduction upper bound, we conclude that either | lg ^ = 0(lg B), 
or \w = Q(VB). Set the degree of the butterfly to satisfy B > w 2 and \gB > lg^. Then, \ = fl(l), 
i.e. t = tt(d). This is intuitive: it shows that the query needs to be as slow as the depth, essentially traversing 
a source to sink path. 

Finally, note that the depth is d = 9(log B 7V). Since \gB > max {2 Igw, lg ^ } = f2(lgw + 
lg §■) = ft(lg ^) . Note that certainly d < w, so lg B = 0(lg ). We obtain i = 17(d) = fi(lg AT/ lg ^). 

6 Proof of the LSD Lower Bounds 

Our goal here is to prove Theorem 21 our optimal lower bound for LSD. 
6.1 The Hard Instances 

We imagine the universe to be partitioned into N blocks, each containing B elements. Alice's set S will contain 
exactly one value from each block. Bob's set T will contain values from each block; more precisely, it will 
contain one value from each pair {(j, 2k); (j, 2k + 1)}. 

Let S and T be the possible choices for S and T according to these rules. Note that |<S| = B N and 
|T| = 2 NB / 2 . We denote by Si Alice's set restricted to block i, and by T Bob's set restricted to block i. Let 
Si and Ti be the possible choices for Si and Ti. We have |<5>i| = B and \%\ — 2 B I 2 . 

We now define T> Y es to be the uniform distribution on pairs (S,T) £ S x T with S n T — 0. In each block 
i, there are two natural processes to generate (Si, Ti) e Si x % subject to SiCiTi = 0: 

1. Pick Ti € Ti uniformly at random, i.e. independently pick one element from each pair {(i, 2k), (i, 2k + 
1)}. Then, pick the singleton Si uniformly at random from the complement of Ti. Note that H(5i | 
T0=log a (B/2). 

2. Pick Si to be a uniformly random element from block i. Then, pick Ti such that it doesn't intersect Si. 
Specifically, if Si fl {2k, 2k + 1} = 0, Ti contains a random element among 2k and 2k + 1. Otherwise, 
^ gets the element not in Note that H(T j; | 5 ? ) = f - 1. 

To generate the distribution T> YES , we will employ the following process. First, pick q £ {0, 1}^ uniformly 
at random. For each qi = 0, apply process 1. from above in block i; for each = 1, apply process 2. in block 
i. Now let Q be a random variable entailing: the vector q; the value Si for every i with qi = 0; and the value 
Ti for every i with qi = 1. Intuitively, Q describes the "first half" of each random process. 

We now define distributions as follows. In block k (called the designated block), choose (Sk,Tk) € 
Sk x Tk uniformly. Notice that Pr[Sk H ^ 0] = |. In all other blocks i ^ k, choose (Si,Ti) E Si x % as 
in the distribution 2? Y es above. As above, we have a vector containing: qi for i ^ k; all such that 

<7i = 0; and all Tj such that qi = 1. 

We are going to prove that: 

Theorem 13. Fix S > 0. //a protocol for LSD has error less than on distribution 53i=i i/ien 
either Alice sends at least SNlgB bits, or Bob sends at least N ■ B 1 ^ ^ bits. 

The distribution 2? Y es will be used to measure various entropies in the proof, which is convenient because 
the blocks are independent. However, the hard distribution on which we measure error is the mixture of 
T>iS. (Since V YES only has yes instances, measuring error on it would be meaningless.) While it may seem 
counterintutive that we argue about entropies on one distribution and error on another, remember that D Y es 
and Di are not too different: S and T are disjoint with probability i when chosen by T>i. 
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6.2 A Direct Sum Argument 



We now wish to use a direct-sum argument to obtain a low-communication protocol for a single subproblcm 
on Si x %. Intuitively, if the LSD problem is solved by a protocol in which Alice and Bob communicate a, 
respectively b bits, we might hope to obtain a protocol for some subproblcm i in which Alice communicates 
O(jf) bits and Bob communicates 0(77) bits. 

Let 7T be the transcript of the communication protocol. If Alice sends a bits and Bob b bits, we claim 
that lv ym {S : tt | Q) < a and Lr> YES (T : ir \ Q) < b. Indeed, once we condition on Q, S and T are independent 
random variables: in each block, either S is fixed and T is random, or vice versa. The independence implies 
that all information about S is given by Alice's messages, and all information about T by Bob's messages. 

Define S<i = (Si, . . . , Sj-i). We can decompose the mutual information as follows: Id ot (S : tt \ Q) = 
J2i=i iDrasCSi : T I Q, S<i). The analogous relation holds for T. By averaging, it follows that for at least half 
of the values of i, we simultaneously have: 

An 4b 
I (Si : 7T I Q, S <t ) < — and I (T, t : tt \ Q, T«) < -. (1) 

iA _ ES iV ^YES -/* 

Remember that the average error on -k ^ T>i is gjLg . Then, there exists k among the half satisfying (JTJ) , 
such that the error on T> k is at most g^jg • For the remainder of the proof, fix this k. 

We can now reinterpret the original protocol for LSD as a new protocol for the disjointness problem in 
block k. This protocol has the following features: 

Inputs: Alice and Bob receive Sk & Sf., respectively Tj, G 

Public coins: The protocol employs public coins to select Q-k- For every i < k with qi = 0, Si is chosen 
publicly to be disjoint from Ti (which is part of Q-k)- For every i < k with q,; = 1, T$ is chosen publicly 
to be disjoint from Si. 

Private coins: Alice uses private coins to select Si for alii > k with qi — 0. Bob uses private coins to select 
Ti for alii > k with = 1. As above, Si is chosen to be disjoint from Tj (which is public knowledge, 
as part of Q-k), and analogously for Tj. 

Error: When Sk and are chosen independently from Sk x Tk, the protocol computes the disjointness of 
Sk and Xfc with error at most g^g- Indeed, the independent choice of Sk and Tk, and the public and 
private coins realize exactly the distribution T>k- 

Message sizes: Unfortunately, we cannot conclude that the protocol has small communication complexity 
in the regular sense, i.e. that the messages are small. We will only claim that the messages have small 
information complexity, namely that they satisfy ((TJ). 

Observe that the disjointness problem in one block is actually the indexing problem: Alice receives a 
single value (as the set Sk) and she wants to determined whether that value is in Bob's set. Since \Sk\ = 1, 
we note that S k f~l T k = iff S k t- Tk- 



6.3 Understanding Information Complexity 

In normal communication lower bounds, one shows that if the protocol communicates too few bits, it must 
make a lot of errors. In our case, however, we must show that a protocol with small information complexity 
(but potentially large messages) must still make a lot of error. 

Let us see what the information complexity of ([l} implies. We have: 

I (Sk ■ tt I <9,5<i) = \- I (Sk:n\q k = l ) T k ,Q-k,S <i ) 

+ 2 ' '■ ^ 1 1^ = 0, Sk,Q-k, S<i) 
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The second term is zero, since H(Sk \ S k ) = 0. Thus, the old bound I-D YES (Sk '■ n \ Q,S<i) < 77 can be 
rewritten as lv YES (Sk ■ tt \ qk = 1, Tfc, Q-k,S<i) < 7^. We will now aim to simplify the left hand side of this 
expression. 

First observe that we can eliminate qk = 1 from the conditioning: Ii> YES (iS l fc : tt | q^ — l,T k ,Q- k , S<i) = 
I-D YES (5fc : tt I T k , Q-k, S<i). Indeed, tt is a function of 5 and T alone. In other words, it is a function of the 
public coins Q-k, the private coins, Sk, and TV But the distribution of the inputs is the same for qk = 1 
and % = 0. In particular, the two processes for generating Sk and Tk (one selected by qk = 0, the other by 
qk = 1) yield the the same distribution. 

Now remember that 2? Y es is simply T>k conditioned on SkC\Tk = 0. Thus, we can rewrite the information 
under the uniform distribution for Sk and Tk: I(S k '■ tt | Q-k, Tk,Sk <£. Tk, S< k ) < 77- (To alleviate notation, 
we drop subscripts for I and H whenever uniform distributions are used.) We are now measuring information 
under the same distribution used to measure the error. 

Analogously, it follows that l(Tk : tt \ Q-k, Sk, Sk <t T k ,T <k ) < f|. We can now apply three Markov 
bounds, and fix the public coins (Q-k, S < k, and T <k ) such that all of the following hold: 

1. the error of the protocol is at most 959-5-; 

2. \(S k :TT\Tk,SktTk) < ^; 

3. l(T k :n\S k ,Sk^T k )<^. 

To express the guarantee of 1., define a random variable £ which is one if the protocol makes an error, 
and zero otherwise. Note that £ is a function £ : Sk x Ca x Tk x Cb — > {0, 1}, where we defined Ca as the set 
of private coin outcomes for Alice and Cb as the private coin outcomes for Bob. By 1., we have E[£] < g^jg- 

We can rewrite 2. by expanding the definition of information: 

l(S k : tt I T k , S k <t Tk) = H(5 fe | T k ,S k </_ T k ) - U(S k \ T k ,n, S k </_ T k ) 
= log 2 f -R(Sk\Tk,Tr,Sk<£T k ) 

Applying a similar expansion to T k , we conclude that: 

log 2 f -R(S k \Tk,n,Sk<£T k ) < ^ (2) 
(f-l)-H(T fc \Sk,TT,S k ^T k ) < ^ (3) 

Consider some transcript tt of the communication protocol. A standard observation in communication 
complexity is that the set of inputs for which tt — tt is a combinatorial rectangle in the truth table of the 
protocol: one side is a subset of S k x Ca, and the other a subset of Tk x Cb- In any rectangle, the output of 
the protocol is fixed. 

Observe that the probability that the output of the protocol is "no" is at most i (the probability that 
Sk and Tk intersect) plus gj^g (the probability that the protocol makes an error). Discard all rectangles on 
which the output is "no." Further discard all rectangles that fail to satisfy any of the following: 

E[£ I TT 

log 2 f -R(S k \T k ,Sk£T k ,TT 
(§-l)-K(T k \Sk,Sk<£T k ,TT 

By the Markov bound, the mass of rectangles failing each one of these tests is at most |. In total, at most 
\ + 99l)9 + 3 • I < 1 of the mass got discarded. Thus, there exists a rectangle tt with answer "yes" that 
satisfies all three constraints. 

Let a be the distribution of Sk conditioned on tt = tt, and r be the distribution of Tk conditioned on 
tt = tt. With this notation, we have: 

1. E^ T [£] < gig, thus Pr^iSk n T k ^ 0] < 9H9. 



"J — 9999 

TT) < 256- 

50 < 2 -W 



13 



2. H CT , T (5 fc | T k ,S k t T k ) > log 2 f - 

3. H CT>T (S fe | T fc , S fc £ T fc ) > (f - 1) - 

In the next section, we shall prove that in every "large enough" rectangle (in the sense of entropy) the 
probability that Sk and Tk intersect is noticeable: 

Lemma 14. Let 7 > 0. Consider probability distributions cr on support Sk, and r on support Tk- The 
following cannot be simultaneously true: 

Pr[^.nT fe ^0] < i (4) 

(TXT 

H (S fc \T k ,S k £T k ) > (l- 7 )log 2 5 (5) 

(TXT 

H (T fe I S fe , Sfc ct T k ) > f - sJ;y i? 1 " 7 ^ (6) 
Since < one of the following must hold: 

logaf < (l-7)log 2 B => a > ^-N\og 2 B 

(B i\ 256b < B 1_ Rl-77 . 1. > 1 at Dl-7-y 

V 2 ^ JV — 2 840 ' -° ^ " — 216000 ' iv ' ° 

For AT and B greater than a constant, it follows that either Alice sends at least SNlgB bits, or Bob must 
send at least 21^0^ • B 1 " 1799 6 bits. 

6.4 Analyzing a Rectangle 

The goal of this section is to show Lemma 1141 Let fi a and /i T be the probability density functions of a 
and r. We define <S* as the set of values of S k that do not have unusually high probability according to a: 
S* = {Sk I ^a{Sk) < l/S 1-77 }. We first show that significant mass is left in <S*: 

Claim 15. ^(S*) > \. 

Proof. Our proof will follow the following steps: 

1. We find a column Tk in which the function is mostly one (i.e. typically S k <f_ T k ), and in which the 
entropy B. a (S k | S k <f- T k ) is large. 

2. The mass of elements outside S* is bounded by the mass of elements outside S* and disjoint from T k , 
plus the mass of elements intersecting T k . The latter is small by point 1. 

3. There are few elements outside S* and disjoint from T k , because they each have high probability. Thus, 
if their total mass were large, their low entropy would drag down the entropy of H a (S k \ Sk (t Tk), 
contradiction. 

To achieve step 1., we rewrite (U]) and ([5]) as: 

1 



< 
- 10 



Pr [S k C T k ] = E Pr[5 fc C T k ] 

(TXT T L (T 

log 2 B - H (S k I T k , S k £ T k ) = E [log 2 B - K{S k \ S k £ T k )] < 7 log 2 B 

(TXT T L CT 

Applying two Markov bounds on Tfc, we conclude that there exists some T& such that: 

Pr[S k C? k ] < K(S k \S k £T k ) > (l-3 7 )log 2 B (7) 

Define a to be the distribution a conditioned on S k <]LT k . 
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With regards to step 2., we can write ^(S*) > 1 - Pr a [S k S* A S k <f_ T k ] - Pr CT [5 fc C f fe ]. The latter 
term is at most j^. In step 3., we will upper bound the former term by g, implying /i CT (iS*) > i. 
For any variable X and event IS, we can decompose: 

H(X) < Pi[E]-R(X\E) + Pr[^B]-H(X|^B) + H b (Pi[E]), (8) 

where H&(-) < 1 is the binary entropy function. We apply this relation to the variable S k under the distrubtion 
<7, choosing S* as our event E. We obtain: 

H(S fc ) < Pr [5*] • H(5 fc | S k G 5*) + Pr [3*1 • H(5 fc | S fe ^ 5*) + 1 

er (7 cr <7 cr 

We have Hs(S k \ S k G 5*) < log 2 -j since there are at most -j choices for S k disjoint from T k . On the other 
hand, H$(S k \ Sk ^ <S*) < (1 — 7 7 ) log 2 B. Indeed, there are at most B 1-77 distinct values outside S* , since 
each must have probability exceeding l/B 1-77 . We thus obtain: 

H(S fe ) < Pr [S*] ■ log 2 f + Pr \&] ■ (1 - 7 7 ) log 2 B + 1 

cr cr cr 

If we had Prj[5*] > h, we would have H.$(Sk) < (1 — 3.5 7 ) log 2 B + 1 < (1 — 3 7 ) log 2 B for large enough 
B. But this would contradict ([7]), which states that Hg(S k ) > (1 — 3 7 ) log 2 B. 

Since a was the distribution a conditioned on S k Tk, Bayes' rule tells us that Pr a [S k ^ S* A Sk <f. 
Tk\<PT d [SkiS*\<\. □ 



Let us now consider the function f(T k ) = E a [\S k C)T k \]. By linearity of expectation, f(T k ) = J2xeT k P f cr[£ € 
Sk] = J2xeT k Mo- 0*0 1 since Sk has a single element. Since \Sk H T k \ G {0, 1}, we can write: 

Pr[S fc nT fc9 6 0] = E[|5 fc nT fe |] = E[E[|5 fc nT fe |]l = E[/(T fc )] 

cr.r cr,r r Lcr J r 

Thus, to reach a contradiction with ([4]), we must lower bound the expectation of /(•) over distribution r. 
Since we do not have a good handle on t, we will approach this goal indirectly: at first, we will completely 
ignore r, and analyze the distribution of f(T k ) when Tk is chosen uniformly at random from Tk- After this, 
we will use the high entropy of t, in the sense of ©, to argue that the behavior on r cannot be too different 
from the behavior on the uniform distribution. 

The expectation of /(•) over the uniform distribution is simple to calculate: Ex fc e"5. [/(!&)] = ~^2 x P^T k eT k [ x ( 
Tk] ■ Ha{x) — ^2 X = \- In the sums, x ranges over elements in block k, each of which appears in Tk 

with probability i. Note that /i a is a probability density function, so J2 X /"a- (a?) = 1- 

Our goal now is to show that when Tk is uniform in Tk, the distribution of /(•) is tightly concentrated 
around its mean of |, and, in particular, away from zero. We will employ a Chernoff bound: we have 
f{Tk) — J2x£T k ^"i x )i an d each x G Tk is chosen independently among two distinct values. Thus, /(Tfc) is 
the sum of B/2 random elements of each chosen independently. 

The limitation in applying the Chernoff bound is the value of max s \i a (x), which bounds the variance of 
each sample. The set S* now comes handy, since we can restrict our attention to elements x with small 
Formally, consider f*(T k ) = J2xeT k nS* Mo-(«)- Clearly f*(T k ) is a lower bound for f(T k ). 

The mean of /*(•) is E TfceTfc [f*{T k )] = J2 xeS , Pr Tk eT k [x G T k ] ■ p a {x) = \^{S*) > ± When T k is 
uniform, f*(T k ) is the sum of B/2 independent random variables, each of which is bounded by l/B 1-77 . 
By the Chernoff bound, 

Pr im)<M] < e-^-^i < e -^/» (9) 

-< fete Ik 

Now we are ready to switch back to distribution r: 
Claim 16. Pr T [/*(T fc ) < i] < \. 
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Proof. The main steps of our proof are: 

1. As in the analysis of S* , we find a row Sk in which the function is mostly one (i.e. typically Sk <jt 2fe), 
and in which the entropy H T (Tk \ Sk <£. Tk) is large. 

2. Pr T [/*(T fe ) < ±] is bounded by Pr T [/*(T fc ) < ^ A T k ], plus the probability that S k C T k . The 
latter is small by point 1. 

3. There are few distinct values of Tk for which f*(T k ) < ^j. If these values had a large mass conditioned 
on Sk (j- Tk, they would drag down the entropy of H T {Tk \ S k <f_ T k ). 

To achieve step 1., we rewrite Q and ^ as: 



Pr [S k C T fe ] = E Pr[S fc C T fc ] 

ffXr r L <7 

f - H (T fc | S fc , S k £T k ) = E [f - H(T fc | S fc £ T, 



< A 



< J_ . #1-77 
- 840 



Applying two Markov bounds on Sk, we conclude that there exists some Sk such that: 



Pr[5 fc cT fc ] < H(T fc | S fc £ T fc ) > | - ^ • B 1 ^ (10) 



Define t to be the distribution r conditioned on Tk- 
For step 2., we can write: 

Pr [/*(T fc ) < i] = Pr [/*(T fc ) <± A SktTk] + Pr [/*(T fc ) < i A 5 fe C T k ] 

< Pr[r(T k )<-&\SktT k ] + Pr [S k cT k ] < Pr[/*(T fc )<i] + A 

We now wish to conclude by proving that Pr^ [/*(!}.) < < i. We apply the relation ([8]) to the variable 
Tk distributed according to r, with the event chosen to be f*(T k ) < 4j: 



Hr{Tk) < Pr [/* {T k ) < &\ ■ ff ? (T* | /* (T fe ) < i) + Pr [/*(T fe ) > 
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By ®, there are at most 2 B / 2 /e Bl '/ 80 distinct choices of T fc such that /*(T fc ) < i. Thus, H ? (T k \ 

If Pr ? [/*(T fc ) < i] > i then ff ? (T fc ) < § - B 1 ^ • ^ + 1 < f - B 1 - 7 V280 for sufficiently large 
But this contradicts (ITOl. □ 



We have just shown that Pr CT>T [5 fc n T fe ^ 0] = E T [/(T fe )] > E T [/*(!&)] > i • § = i. This contradicts 
(gj. Thus, at least one of (gj), (0, and © must be false. 

This concludes the proof of Lemma [14] and of Theorem [131 



7 Conclusion 

We have shown that many important lower bounds can be derived from a single core problem, through a 
series of clean, conceptual reductions. It is unclear what the ultimate value of this discovery will be, but the 
following thoughts come to mind: 

1. We are gaining understanding into the structure of the problems at hand. 

2. We simplify several known proofs. For example, we sidestep the technical complications in the previous 
lower bounds for 2D range counting [40] and exact nearest neighbor [11) . 

3. We can now teach data-structure lower bounds to a broad audience. Even "simple" lower bounds 
are seldom light on technical details. By putting all the work in one bound, we can teach many interesting 
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results through clean reductions. (If we are satisfied with deterministic bounds, the lower bound for set 
disjointness from [36] is a one-paragraph counting argument.) 

4. Our results hint at a certain degree of redundancy in our work so far. In doing so, they also mark the 
borders of our understanding particularly well, and challenge us to discover surprising new paths that go far 
outside these borders. 

Acknowledgements. The author would like to thank Yakov Nekrich and Marek Karpinsky for useful 
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References 

Miklos Ajtai. A lower bound for finding predecessors in Yao's cell probe model. Combinatorica, 8(3):235- 
247, 1988. 

Stephen Alstrup, Amir M. Ben-Amram, and Theis Rauhe. Worst-case and amortised optimality in 
union-find. In Proc. 31st ACM Symposium on Theory of Computing (STOC), pages 499-506, 1999. 

Stephen Alstrup, Gerth S. Brodal, Inge Li G0rtz, and Theis Rauhe. Time and space efficient multi- 
method dispatching. In Proc. 8th Scandinavian Workshop on Algorithm Theory (SWAT), pages 20-29, 
2002. 

Stephen Alstrup, Gerth S. Brodal, and Theis Rauhe. New data structures for orthogonal range searching. 
In Proc. 41st IEEE Symposium on Foundations of Computer Science (FOCS), pages 198-207, 2000. 

Stephen Alstrup, Thore Husfeldt, and Theis Rauhe. Marked ancestor problems. In Proc. 39th IEEE 
Symposium on Foundations of Computer Science (FOCS), pages 534-543, 1998. 

Stephen Alstrup, Thore Husfeldt, and Theis Rauhe. A cell probe lower bound for dynamic nearest- 
neighbor searching. In Proc. 12th ACM/SIAM Symposium on Discrete Algorithms (SODA), pages 
779-780, 2001. 

Alexandr Andoni, Dorian Croitoru, and Mihai Patra§cu. Hardness of nearest-neighbor search under 
1-infinity. In Proc. 49th IEEE Symposium on Foundations of Computer Science (FOCS), 2008. 

Alexandr Andoni, Piotr Indyk, and Mihai Patra§cu. On the optimality of the dimensionality reduction 
method. In Proc. 41th IEEE Symposium on Foundations of Computer Science (FOCS), pages 449-458, 
2006. 

David Applegate, Gruia Calinescu, David S. Johnson, Howard J. Karloff, Katrina Ligett, and Jia Wang. 
Compressing rectilinear pictures and minimizing access control lists. In Proc. 18th ACM/SIAM Sym- 
posium on Discrete Algorithms (SODA), pages 1066-1075, 2007. 

Ziv Bar-Yossef, T. S. Jayram, Ravi Kumar, and D. Sivakumar. An information statistics approach to 
data stream and communication complexity. Journal of Computer and System Sciences, 68(4):702-732, 
2004. See also FOCS'02. 

Omer Barkol and Yuval Rabani. Tighter lower bounds for nearest neighbor search and related problems 
in the cell probe model. Journal of Computer and System Sciences, 64(4):873-896, 2002. See also 
STOC'00. 

Paul Beame and Faith E. Fich. Optimal bounds for the predecessor problem and related problems. 
Journal of Computer and System Sciences, 65(l):38-72, 2002. See also STOC'99. 



17 



[13] Amir M. Bcn-Amram and Zvi Galil. A generalization of a lower bound technique due to Fredman and 
Saks. Algorithmica, 30(l):34-66, 2001. See also FOCS'91. 

[14] Allan Borodin, Rafail Ostrovsky, and Yuval Rabani. Lower bounds for high dimensional nearest neighbor 
search and related problems. In Proc. 31st ACM Symposium on Theory of Computing (STOC), pages 
312-321, 1999. 

[15] Amit Chakrabarti, Bernard Chazelle, Benjamin Gum, and Alexey Lvov. A lower bound on the complex- 
ity of approximate nearest- neighbor searching on the hamming cube. In Proc. 31st ACM Symposium 
on Theory of Computing (STOC), pages 305-311, 1999. 

[16] Amit Chakrabarti and Oded Regev. An optimal randomised cell probe lower bound for approximate 
nearest neighbour searching. In Proc. 45th IEEE Symposium on Foundations of Computer Science 
(FOCS), pages 473-482, 2004. 

[17] Moses Charikar, Piotr Indyk, and Rina Panigrahy. New algorithms for subset query, partial match, 
orthogonal range searching, and related problems. In Proc. 29th International Colloquium on Automata, 
Languages and Programming (ICALP), pages 451-462, 2002. 

[18] Bernard Chazelle. A functional approach to data structures and its use in multidimensional searching. 
SI AM Journal on Computing, 17:427-462, 1988. Sec also FOCS'85. 

[19] Richard Cole, Lee- Ad Gottlieb, and Moshe Lewenstein. Dictionary matching and indexing with errors 
and don't cares. In Proc. 36th ACM Symposium on Theory of Computing (STOC), pages 91-100, 2004. 

[20] Mark de Berg, Marc J. van Kreveld, and Jack Snoeyink. Two- and three-dimensional point location in 
rectangular subdivisions. Journal of Algorithms, 18(2):256-277, 1995. See also SWAT'92. 

[21] David Eppstcin and S. Muthukrishnan. Internet packet filter management and rectangle geometry. In 
Proc. 12th ACM/SIAM Symposium on Discrete Algorithms (SODA), pages 827-835, 2001. 

[22] Anja Feldmann and S. Muthukrishnan. Tradeoffs for packet classification. In Proc. IEEE INFOCOM, 
pages 1193-1202, 2000. 

[23] Paolo Ferragina and S. Muthukrishnan. Efficient dynamic method-lookup for object oriented languages. 
In Proc. 4th European Symposium on Algorithms (ESA), pages 107-120, 1996. 

[24] Paolo Ferragina, S. Muthukrishnan, and Mark dc Berg. Multi-method dispatching: A geometric ap- 
proach with applications to string matching problems. In Proc. 31st ACM Symposium on Theory of 
Computing (STOC), pages 483-491, 1999. 

[25] Michael L. Fredman and Monika Rauch Henzinger. Lower bounds for fully dynamic connectivity prob- 
lems in graphs. Algorithmica, 22(3):351-362, 1998. 

[26] Michael L. Fredman and Michael E. Saks. The cell probe complexity of dynamic data structures. In 
Proc. 21st ACM Symposium on Theory of Computing (STOC), pages 345-354, 1989. 

[27] Anna Gal and Peter Bro Miltersen. The cell probe complexity of succinct data structures. In Proc. 30th 
International Colloquium on Automata, Languages and Programming (ICALP), pages 332-344, 2003. 

[28] Thore Husfeldt and Theis Rauhe. New lower bound techniques for dynamic partial sums and related 
problems. SIAM Journal on Computing, 32(3):736-753, 2003. See also ICALP'98. 

[29] Thore Husfeldt, Theis Rauhe, and S0ren Skyum. Lower bounds for dynamic transitive closure, planar 
point location, and parentheses matching. In Proc. 5th Scandinavian Workshop on Algorithm Theory 
(SWAT), pages 198-211, 1996. 



18 



[30] Piotr Indyk. On approximate nearest neighbors under norm. Journal of Computer and System 
Sciences, 63(4):627-638, 2001. See also FOCS'98. 

[31] T. S. Jayram, Subhash Khot, Ravi Kumar, and Yuval Rabani. Cell-probe lower bounds for the partial 
match problem. Journal of Computer and System Sciences, 69(3):435-447, 2004. See also STOC'03. 

[32] Bala Kalyanasundaram and Georg Schnitger. The probabilistic communication complexity of set inter- 
section. SIAM Journal on Discrete Mathematics, 5(4):545-557, 1992. See also Structures'87. 

[33] Ding Liu. A strong lower bound for approximate nearest neighbor searching. Information Processing 
Letters, 92(l):23-29, 2004. 

[34] Peter Bro Miltcrsen. The bit probe complexity measure revisited. In Proc. 10th Symposium on Theo- 
retical Aspects of Computer Science (STACS), pages 662-671, 1993. 

[35] Peter Bro Miltersen. Lower bounds for Union-Split-Find related problems on random access machines. 
In Proc. 26th ACM Symposium on Theory of Computing (STOC), pages 625-634, 1994. 

[36] Peter Bro Miltersen, Noam Nisan, Shmuel Safra, and Avi Wigderson. On data structures and asym- 
metric communication complexity. Journal of Computer and System Sciences, 57(l):37-49, 1998. See 
also STOC'95. 

[37] Peter Bro Miltersen, Sairam Subramanian, Jeffrey S. Vittcr, and Roberto Tamassia. Complexity models 
for incremental computation. Theoretical Computer Science, 130(l):203-236, 1994. See also STACS'93. 

[38] S. Muthukrishnan and Martin Miiller. Time and space efficient method-lookup for object-oriented 
programs. In Proc. 7th ACM/SIAM Symposium on Discrete Algorithms (SODA), pages 42-51, 1996. 

[39] Yakov Nekrich. A data structure for multi-dimensional range reporting. In Proc. 23rd ACM Symposium 
on Computational Geometry (SoCG), pages 344-353, 2007. 

[40] Mihai Patra§cu. Lower bounds for 2-dimensional range counting. In Proc. 39th ACM Symposium on 
Theory of Computing (STOC), pages 40-46, 2007. 

[41] Mihai Patra§cu and Erik D. Demaine. Logarithmic lower bounds in the cell-probe model. SIAM Journal 
on Computing, 35(4):932-963, 2006. See also SODA'04 and STOC'04. 

[42] Mihai Patra§cu and Corina Tarni^a. On dynamic bit-probe complexity. Theoretical Computer Science, 
380:127-142, 2007. Sec also ICALP'05. 

[43] Mihai Patra§cu and Mikkel Thorup. Higher lower bounds for near-neighbor and further rich problems. 
In Proc. 47th IEEE Symposium on Foundations of Computer Science (FOCS), pages 646-654, 2006. 

[44] Mihai Patra§cu and Mikkel Thorup. Time-space trade-offs for predecessor search. In Proc. 38th ACM 
Symposium on Theory of Computing (STOC), pages 232-240, 2006. 

[45] Mihai Patra§cu and Mikkel Thorup. Randomization does not help searching predecessors. In Proc. 18th 
ACM/SIAM Symposium on Discrete Algorithms (SODA), pages 555-564, 2007. 

[46] Alexander A. Razborov. On the distributional complexity of disjointness. Theoretical Computer Science, 
106(2):385-390, 1992. 

[47] Ronald L. Rivest. Partial-match retrieval algorithms. SIAM Journal on Computing, 5(l):19-50, 1976. 
See also FOCS'74 and Stanford PhD thesis. 

[48] Pranab Sen and Srinivasan Venkatesh. Lower bounds for predecessor searching in the cell probe model. 
Journal of Computer and System Sciences, 74(3):364-385, 2008. See also ICALP'01, CCC'03. 



19 



[49] Mikkel Thorup. Space efficient dynamic stabbing with fast queries. In Proc. 35th ACM Symposium on 
Theory of Computing (STOC), pages 649-658, 2003. 

[50] Peter van Emde Boas, R. Kaas, and E. Zijlstra. Design and implementation of an efficient priority 
queue. Mathematical Systems Theory, 10:99-127, 1977. Conference version by van Emde Boas alone in 
FOCS'75. 

A Catalog of Problems 

Range Queries. Given a set of n queries in d-dimensional space (say, [n] d ), we can ask two classic queries: 
report the points inside a range [ai,&i] x • • • x [0,2,6,2], or simply count the number of points in the range. 
These queries lie at the heart of database analysis, and any course on SQL is bound to start with an example 
of the form: "find employees born between 1980 and 1989, whose salary is between $80,000 and $90,000." 

Note that if there are k points inside the range, reporting them necessarily takes time f2(fc). To avoid 
this technicality, in this paper we only consider the decision version of reporting: is there a point inside the 
range? 

Stabbing queries. A dual of range queries is stabbing: preprocess a set of n boxes of the form [ai, &i] x 
• • • x [ad, bd], such that we can quickly find the box(es) containing a query point. 

Stabbing is a very important form of classification queries. For instance, network routers have rules 
applying to packets coming from some IP range, and heading to another IP range. A query is needed for 
every packet passing through the router, making this a critical problem. This application has motivated 
several theoretically-minded papers [I§1 [2H HI IS] , as well as a significant body of practically- minded ones. 

Another important application of stabbing is method dispatching, in experimental object oriented lan- 
guages that (unlike, say, Java and C++) allow dynamic dispatching on more arguments than the class. 
This application has motivated several theoretically-minded papers [38l [3] [23] [24] , as well as a number of 
practically-minded ones. 

Our lower bounds for 2D stabbing are the first for this problem, and in fact, match the upper bound 

of ng. 

It is easy to see that stabbing in d dimensions reduces to range reporting in 2d dimensions, since boxes 
can be expressed as 2d-dimensional points. 

The decision version of stabbing in 2D also reduces to (weighted) range counting in 2D by the following 
neat trick. We replace a rectangle [ai,6i] x [02,62] by 4 points: (ai,6i) and (02,62) with weight +1, and 
(01,62) and (02,61) with weight — 1. To test whether (91,92) stabs a rectangle, query the sum in the range 
[0, qi] x [0, q-2\. If the query lies inside a rectangle, the lower-left corner contributes +1 to count. If the query 
point is outside, the corners cancel out. 

With a bit of care, the reduction can be made to work for unweighted range counting, by ensuring the 
query never stabs more than one rectangle. Then, it suffices to count points mod 2. 

Partial match. The problem is to preprocess a data base of n strings in {0, l} d . Then, a query string 
from the alphabet {0,1, *} d is given, and we must determine whether any string in the database matches 
this pattern (where * can match anything). This is equivalent to a problem in which the query is in {0, l} d , 
and we must test whether any string in the database is dominated by the query (where o dominates 6 if on 
every coordinate a, > 6,). 

The first upper bounds for partial match was obtained by Rivest [47], who showed that the trivial 2 d 
space can be slightly improved when d < 21gn. Charikar, Indyk, and Panigrahy |17) showed that query 

time 0(n/2 T ) can be achieved with space n ■ 2°( dlg d /V T / l % n \ It is generally conjectured that the problem 
follows from the curse of dimensionality, in the following sense: there is no constant e > 0, such that query 
time 0(n 1_e ) can be supported with space poly(m) • 2 oi - d € \ 
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If the problem is parameterized by the number of stars k, it is trivial to achieve space 0(n) and query 
time 0(2 k ) by exploiting the binary alphabet. In the more interesting case when the alphabet can be large, 
Cole, Gottlieb, Lewenstein jT5] achieve space 0(n \g k n) and time 0(\g k n ■ lglgn) for any constant k. 

Partial match can be reduced [30] to exact near neighbor in l\ or £2, and to 3-approximate near neighbor 
in ioo. This is done by applying the following transformation to each coordinate of the query: 1— > — ^; 

Marked ancestor. In this problem, defined by Alstrup, Husfeldt, and Rauhe [5], we are to maintain a 
complete tree of degree b and depth d, in which vertices have a mark bit. The updates may mark or unmark 
a vertex. The query is given a leaf v, and must determine whether the path from the root to v contains any 
marked node. In our reduction, we work with the version of the problem in which edges are labeled, instead 
of nodes. However, note that the problems are identical, because we can attach the label of an edge to the 
lower endpoint. 

Marked ancestor reduces to dynamic stabbing in ID, by associating each vertex with an interval extending 
from the leftmost to the rightmost leaf in its subtree. Marking a node adds the interval to the set, and 
unmarking removes it. Then, an ancestor of a leaf is marked iff the leaf stabs an interval currently in the 
set. 

The decremental version, in which we start with a fully marked tree and may only unmark, can be 
reduced to union-find. Each time a node is unmarked, we union it with its parent. Then, a root-to-leaf path 
contains no marked nodes iff the root and the leaf are in the same set. 

The lower bounds of this paper work for both the decremental and incremental variants. 
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